On the 12th Day of Christmas, a Statistician Sent to Me…

Riley, R. D., Cole, T. J., Deeks, J., Kirkham, J. J., Morris, J., Perera, R. et al. (2022). On the 12th Day of Christmas, a Statistician Sent to Me. BMJ, 379, e072883.

The BMJ statistical editors 2022 list of the 12 most common statistical faux pas that they encounter in reviewing articles for submission to the journal (I am guilty of most of these myself).

Being proactive about these issues will not only make you better clinical researchers, it will also increase your chances of a successful submission (to any journal).

On the 1st day of Christmas, a statistician sent to me:

Clarify the research question

So, here we go. On the 1st day of christmas, a statistician sent to me:

Clarify the research question

Now, these may seem like some very basic ideas but you’d be surprised how often researchers just haven’t put enough thought into this, and then they get flustered when they’re put on the spot.

You need to be be able to clearly articulate your research question - and I’d suggest being able to do that in both a sciency way and in a lay way, because you may need to present to different audiences. And as part of that you also need to have thought about how you will answer your question in the structured way that your more specific aims and objectives should lay out.

Then, moving more towards the analysis, you need to be able to say, what is the intent of your research. Is it meant to describe something - for example the prevalence of MS in a particular population - in which case you’ll mainly be interested in summarising your data and using descriptive statistics.

Or is it meant to explain something - an example here might be to assess whether some environmental factor might be related to the development of MS. Here you could look simply at whether there was an ‘association’ between the environmental factor and MS risk, but with the caveat that there could be some unmeasured confounding still at play. Because let’s face it - most of our work in observational research is about trying to quantify associations rather than assert causation, because we can never be sure about unmeasured confounding. If you want to go the next step and show causation - that the factor causes MS - that’s a much more complex analysis requiring a much more restrictive set of assumptions. In any case, inferential statistics is what you’ll want here.

Alternatively you may be interested more in predicting an outcome, rather than explaining relationships. And in this case, you might not even care about inference - you just want to maximise predictive power.

By now you should be clear about what your outcome is, but you also need to make sure that being able to estimate that outcome aligns well with being able to answer your research question.

https://www.discoverphds.com/advice/doing/research-aims-and-objectives Shmueli, G. (2010). To Explain or to Predict. Statistical Science, 25(3), 289-310.

What is your research question (aims/objectives)?
Is your research meant to :
- Describe
  - Summarise the data.
  - Focus on descriptive statistics.
    - Means (± SDs), Medians (IQR), Counts (%s).
- Explain (Why?)
  - Association vs Causation.
  - Focus on inferential statistics.
    - t-tests, \(\chi^{2}\)tests, statistical models.
- Predict (You don’t care about the ‘Why?’)
  - Predict outcome from set of covariates.
  - Focus on maximising predictive power at expense of explanation.
    - Statistical models ± inference.
What is the outcome (described statistically as the ‘estimand’)?
Does knowing the outcome address your research question.
- i.e. is the answer aligned with the question?

On the 2nd day of Christmas, a statistician sent to me:

Focus on estimates, confidence intervals, and clinical relevance

Day 2 - Focus on estimates, confidence intervals and clinical relevance.

We should all know by now that statistical significance DOES NOT equate to clinical significance.

As your sample size increases, even smaller effects can become statistically significant and that’s because they are estimated more precisely. Standard errors - which inform your p value - are based on your sample size - that’s just how statistics works.

So, as this example illustrates, you could have a HR of 0.97 and a p value that’s less than 0.05, but you would have to ask yourself if that 3% reduction in the risk of the outcome is actually clinically important - it may be, but the clinical context needs to be considered.

Conversely, absence of evidence DOES NOT equate to evidence of absence.

You may have a small sample size which leads to clinically important effects being ignored when maybe they shouldn’t be.

In this case a HR of 0.7 means there’s a 30% reduction in the risk of the outcome but the p value is greater than 0.05 - so non-significant. That doesn’t mean that we should ignore the effect though. It could simply be that we don’t have enough statistical power to say with certainty that the observed effect is a true effect in the population, rather than just a chance finding in our particular sample of data. So again, the clinical context needs to be considered.

Remember that the point estimate is the best guess of the population parameter of interest. And the 95% CI provides a range of values for the true estimate of that parameter that are consistent with our data.

So, the idea here is NOT to just look at the p value alone. Consider the point estimate and the 95% CI first - the p value is optional.

Statistical significance \(\neq\) clinical significance.
- If your sample size is large enough, even small effects are statistically significant!
  - e.g. HR = 0.97 (95% C.I. 0.95, 0.99; p < 0.05).
    - Is this still important clinically? (maybe it is).

Conversely:

Absence of Evidence \(\neq\) Evidence of Absence.
- If your sample size is small, clinically relevant effects may be ignored when they shouldn’t be!
  - e.g. HR = 0.70 (95% C.I. 0.40, 1.10; p > 0.05).
    - Is this still important clinically? (maybe it is).

Remember:

The point estimate is the ‘best guess’ of the population parameter of interest.
The 95% CI provides a range of values for the true estimate of the population parameter that are consistent with our data.

On the 3rd day of Christmas, a statistician sent to me:

Carefully account for missing data

Day 3 - Carefully account for missing data

Missing values are a fact of life in clinical research. Best practice means that we should be transparent about the missingness in our data. Quantify and report it - preferably at an individual variable level and describe if you accounted for it some way in your analysis.

There are 3 primary missing data mechanisms - but this is mostly theoretical, because it’s actually not easy to identify the specific mechanism you have, in the first place. In any case if you consider an example where we want to model the association between self-reported body-weight (which is the outcome - or an unobserved variable) and sex (which is the predictor - or an observed variable) - and we have some responses that aren’t completed.

According to this missing data naming scheme, that could occur for one of three reasons:

some individuals just didn’t see the question (Missing Completely at Random). So it doesn’t matter whether you’re male or female, obese or non-obese, the missingness was completely random.
females in general were less willing to reveal their weight than males (Missing at Random). Here the missingness in the unobserved variable - bodyweight - depends on the value of sex. In other words we know what we don’t know.
obese individuals in general were less willing to reveal their weight (Missing Not at Random). Here the missingness in the unobserved variable - bodyweight - depends on the value of bodyweight itself. In other words we don’t know what we don’t know.

What to do about missing data?

Well you could just ignore it - that’s called complete case analysis and that is only unbiased when your missing data are MCAR.

You could impute the missing data - essentially replacing the values with best guesses. Single imputation methods tend to be biased. Multiple imputation is the gold standard, but that’s only unbiased under MCAR/MAR assumptions - and we mostly just assume that’s the case because it’s the best option we have. Multiple imputation basically creates several complete versions of the data by replacing the missing values by plausible data values - the analysis is then performed in each imputed dataset, the parameters pooled into one estimate and the uncertainty in the pooled estimate calculated.

When should you use multiple imputation - well, anytime you have missingness I guess. Really there’s no lower limit to the amount of missingness in order to use MI, but there probably is an upper limit and that’s dependent on your data. You may have 80% missingness in just one variable but if all the other variables are largely intact, you could impute that ok I think.

There’s no doubt we and could and should be using MI more, but it does add considerable complexity to your analysis, and I think that’s probably a big reason for why we don’t see it being applied more in the literature.

Johansson, Å. M., & Karlsson, M. O. (2013). Comparison of methods for handling missing covariate data. AAPS J, 15(4), 1232-1241. Kang, H. (2013). The prevention and handling of the missing data. Korean J Anesthesiol, 64(5), 402-406. https://www.publichealth.columbia.edu/research/population-health-methods/missing-data-and-multiple-imputation https://stats.stackexchange.com/questions/149140/how-much-missing-data-is-too-much-multiple-imputation-mice-r

Missing values are the norm, not the exception.
Quantify and report the amount of missing data.
Explain how the missingness was handled in the analyses.
Missing data mechanisms/assumptions:
- e.g. In your study you wish to model the association between self-reported body weight (Y) and sex (X). Some responses aren’t complete, because:
  - some individuals simply didn’t see the question (Missing Completely at Random [MCAR] - “What you hope for”).
  - females in general were less willing to reveal their weight than males (Missing at Random [MAR] - “Second best”).
  - obese individuals in general were less willing to reveal their weight (Missing Not at Random [MNAR] - “The worst”).
- Very hard to identify the mechanism in any given dataset.
Missing data approaches:
- Ignore.
  - Complete case analysis - unbiased only under MCAR.
- Impute.
  - Single:
    - Mean/Median substitution - biased.
    - Last observation carried forward (LOCF) - biased.
  - Multiple - unbiased under MCAR/MAR.
How little and how much?
- MI with as little as 5% missingness.
- Upper limit depends on data - if you have many variables with minimal missingness, one with 80% missing should theoretically be ok with MI.

On the 4th day of Christmas, a statistician sent to me:

Do not dichotomise continuous variables

Day 4 - do not dichotomise continuous variables

Really this one is quite simple and it shouldn’t be open to interpretation. You deserve to go to the naughty corner behind the christmas tree if you do this.

There are several issues with dichotomising continuous variables - let’s take the example of dichotomising systolic blood pressure at 130 mmHg to illustrate some of these.

The first problem is that you waste information on between-subject variability. Consider two people with BP’s of 129 and 131 mmHg. Using a cut-off of 130 mmHg means the analysis considers those two people different when in fact they are really the same. Conversely, two people with BP’s of 131 and 220 are considered the same in the analysis when they really are different.

Another problem is that you lose statistical power.

If you are going to dichotomise, where do you make the cut? This is ripe for data-dredging and p-hacking if you choose to select the cut point based on statistical significance.

If the relationship of the continuous covariate with the outcome is non-linear, then you will certainly never know that by dichotomising it.

And finally you can end up with residual confounding, when you might have been able to eliminate that completely if you left the confounder in it’s continuous form.

Altman, D. G., & Royston, P. (2006). Statistics Notes: The cost of dichotomising continuous variables. BMJ, 332(7549), 1080. Cumberland, P. M., Czanner, G., Bunce, C., Doré, C. J., Freemantle, N., García-Fiñana, M. et al. (2014). Ophthalmic statistics note 3: the perils of dichotomising continuous variables. Br J Ophthalmol, 98(6), 841-843.

There is no good reason to do this, so don’t do it.
e.g. Dichotomise systolic blood pressure at 130 mmHg.
Wastes information on between-subject variability.
- 129 mmHg and 131 mmHg are considered different when really the same.
- 131 mmHg and 220 mmHg are considered the same when really different.
Loss of statistical power.
- Need a larger sample size to maintain statistical power equivalent to that for continuous data.
Where to dichotomise?
- Can lead to data dredging and the ‘selection’ of optimal cut-points to maximise statistical significance.
Non-linear relationships may be masked by dichotomisation.
Dichotomisation of confounders does not fully correct for confounding of an exposure-outcome association -> residual confounding.

On the 5th day of Christmas, a statistician sent to me:

Consider non-linear relationships

Day 5 - Consider non-linear relationships

So we’re taught linear regression in a way that makes us think that all continuous covariates have a linear relationship with the outcome of interest. But, in reality, many functional forms are non-linear.

Assuming linearity is nice and convenient - we are able to say in our results that a one-unit change in x equals such-and-such a change in y, and that applies across the entire range of covariate values. So, as an example, let’s assume the association between age and the log risk of relapse is linear - we can then say the HR for relapse is the same for someone at 30 as for someone at 90.

But if the association is in fact non-linear, that means that the “one-unit change” spiel is no longer valid, as the effect of the covariate on the outcome will depend on the value of the covariate itself. And in this case what that would mean is that the HR for relapse will be different for a 30 year old compared to that for a 90 year old.

So, how do you model non-linearity. Well, there are several options:

If you have an overall non-linear form, but segments that appear linear within that, you could use linear splines, or piecewise or segmented linear regressions, as they’re also known.

You could use polynomial to model curvature. A disadvantage is that they exert ‘global’ effects - so, one point can influence the entire curve.

You could use fractional polynomials - I won’t go into those.

My go to are restricted cubic splines - these model non-linearity very nicely and are not hard to include in a model.

The primary issue with allowing functional forms to be non-linear is their interpretability. Really the best advice here is to plot the model predictions and assess the relationships visually. You can then calculate effects at specific covariate values.

This may seem like an effort but you need to keep in mind that if the relationship is truly non-linear then you’re just putting lipstick on a pig by pretending otherwise. So we need to keep an open mind to the possibility of non-linear relationships.

Read up on how RCS’s work.

Continuous covariates may have linear OR non-linear association with the outcome.
- Assuming linearity is a stronger assumption and is often wrong.
- More realistic to assume non-linearity as a starting point and test if this is necessary.
A linear association assumes that a 1 unit increase in the covariate has the same ‘effect’ on the outcome across the range of covariate values.
- e.g. HR for risk of relapse for change in age from 30 -> 31, same as for 90 -> 91.
- Provides a simple interpretation.
A non-linear association allows the ‘effect’ of the covariate to vary across the range of covariate values.
- HR for risk of relapse for change in age from 30 -> 31, differs compared to change in age for 90 -> 91.
How to model non-linearity:
- Linear splines - different linear functions across range of covariate values.
  - Better than assuming complete linearity but still restrictive.
- Regular polynomials - quadratics (\(x^2\)), cubics (\(x^3\)), etc (integer powers).
  - Disadvantage - ‘global’ effects. One point can influence the entire curve.
- Fractional polynomials - fractional powers (e.g. \(x^{2/3}\))
- Restricted cubic splines (RCS) - multiple regular cubic polynomials joined smoothly at ‘knots’.
  - Models non-linearity nicely. Can capture local effects without one point influencing the entire curve.
  - Disadvantage - interpretation.
RCS interpretation:
- Visualise - plot model predictions.
- Consider specifying effect at salient values of covariate.
  - e.g. HR for one-year change in age at 30 yrs, vs HR for one-year change in age at 90 yrs.

On the 6th day of Christmas, a statistician sent to me:

Quantify differences in subgroup results

Day 6 - Quantify differences in subgroup results.

This one applies more to clinical trials really, but it is also important to keep in mind for observational research as well.

So sometimes you may be interested in estimating a treatment effect in different subgroups of your data. You’ve worked out a treatment effect in your overall data but you want to know whether that differs across some subgroup - for example, what is the treatment effect amongst males compared to females?

There are essentially two ways you can do this:

The first way is to split your data into the subgroups and run the analysis separately in each. This kind of gets you to an answer but it’s not the recommended approach. One issue is that by splitting your data you lose statistical power and increase the risk of both false positives and false negatives. Another problem is that while you will get a separate treatment effect in each subgroup you have no way of comparing those effects statistically.

The second way - and the correct way to do this - is to run your single model on your overall data but include a statistical interaction term between your treatment and subgroup variables. This will give you a statistical test for whether the treatment effect differs and it will quantify that difference in the process.

The point in all of this is that splitting your data and calculating separate effects tells you nothing about whether those effects actually differ. You need an interaction term to do that for you.

I won’t go through that example that I’ve listed - you can read that on the blog later if you want.

++++++++++++++++++++++++++++++

Plan:

In the first instance always do the interaction test and not split the data for subgroup analyses.
If the interaction is significant (i.e. treatment effect differs across subgroups) then go on and estimate each specific treatment effect (either post-hoc contrasts OR split the data and analyse separately).
If the interaction is N.S. - then each group has the same treatment effect. Drop the interaction and just include as a confounder to adjust for.

Read references 36-38 from the paper: Brankovic, M., Kardys, I., Steyerberg, E. W., Lemeshow, S., Markovic, M., Rizopoulos, D. et al. (2019). Understanding of interaction (subgroup) analysis in clinical trials. Eur J Clin Invest, 49(8), e13145.

Sometimes you may be interested in estimating a treatment (or some other) ‘effect’ in subgroups (sex, disease status, etc).
There are two ways to do this:
- One model with an interaction term between your treatment and subgroup variables (e.g. treatment * sex).
  - -> “The effect of treatment depends on sex”.
- Two models (one for each sex).
  - -> Separate treatment effects.
The first is the better approach.
Consider example - Association between hormone replacement therapy (HRT) and non-vertebral fractures in women.
- In subgroup women < 60 yrs, RR of fracture (HRT vs none) = 0.67 (95% C.I. 0.46, 0.98; p = 0.03).
- In subgroup women ≥ 60 yrs, RR of fracture (HRT vs none) = 0.88 (95% C.I. 0.71, 1.08: p = 0.22).
Naive interpretation is to suggest HRT is beneficial for younger women but not for older women.
BUT, knowing the treatment effect in each group, tells you NOTHING about whether the treatment effects themselves, differ!
The interaction term quantifies this difference.
- Ratio of risk ratio’s = 0.76 (95% C.I. 0.49, 1.17; p = 0.2)
- Another Absence of Evidence \(\neq\) Evidence of Absence problem.
- Older women may still benefit from HRT but we can’t say with certainty - possible loss of power from splitting the data and reducing the sample size.
- The current data don’t support that conclusion.
It is not sufficient for the treatment effect to be significant in one subgroup and not the other - the interaction term must also be significant.

On the 7th day of Christmas, a statistician sent to me:

Consider accounting for clustering

Day 7 - Consider accounting for clustering

Don’t ignore clusters in your data. These are groups of observations that share common characteristics - in other words are correlated - and they can arise in one of two main ways. The first are observations that are correlated within individuals - these are repeated measures on the same person. The second are observations that are correlated across or between individuals - for example, patients in a GP clinic or hospital. You might have data from multiple hospitals and it’s quite possible that two patients from the same hospital may be more alike for various reasons - sociodemographics for example, then two patients from two different hospitals.

Statistical tests are based on an assumption of independent observations, where each observation provides new information and increases statistical power.

This is why as a general rule, you would prefer fewer observations from more people than more observations from fewer people. Measuring EDSS once on 10 patients is going to provide you with more statistical power than measuring EDSS 10 times on one patient.

Statistical tests and models don’t know when observations are correlated - unless you tell them, so they will otherwise treat every observation as unique and independent. This ‘tricks’ them into thinking we have more information than we really do. And this results in standard errors and p values that are typically underestimated - that is, too low - and that results in false positive findings.

The solution is to make sure you’re using modelling approaches that account for the non-independence and these include mixed-effects models and generalised estimating equations.

Don’t ignore clusters in your data - within- or between-individual.
Two main cases:
- Repeated measurements within patients.
- Patients within larger groupings (e.g.hospitals/GP clinics).
These give rise to correlated observations which can bias inference if not accounted for correctly.
Statistical tests/models are based on an assumption of independent observations - each observation provides ‘new’ information.
This is the assumption of independent observations.
If you had to choose between the following 10 observations, which would you choose to maximise the statistical power in your study:
- EDSS measured 10 times on one patient.
- EDSS measured once on each of 10 patients.
If observations are correlated in some way, they no longer provide completely new information but are still treated as such.
This ‘tricks’ our statistical tests/models into thinking we have more information than we really do and may bias inference - usually in a false-positive kind of way (i.e. standard errors and p values erroneously low)
Solution - use an analytical approach that accounts for the non-independence:
- Mixed-models.
- Generalised estimating equations.

On the 8th day of Christmas, a statistician sent to me:

Interpret \(I^2\) and meta-regression appropriately

Day 8 - Interpret I2 and meta-regression appropriately

Not sure how much these are really relevant, but some of you may be doing meta-analyses.

So, in meta-analyses, the idea is to combine multiple smaller studies to increase statistical power in estimating an effect of interest.

Ideally, all the studies being combined would be identical in how they were conducted - we could then say that any observed differences would be due to chance and the studies are considered homogenous.

But that’s not the real world. Studies are never done the same way and therefore we expect some heterogeneity arising from the variability in outcomes beyond that due to chance.

Then the question is - if between-study heterogeneity is high, should we even combine them in the first place?

Now a common measure of heterogeneity is the I2 index. It describes the proportion of the variance in the effect estimate that is due to heterogeneity rather than chance.

If it’s closer to 0%, studies are more homogenous, but if closer to 100%, studies are more heterogenous.

The important point is that it’s a relative measure - not an absolute measure. It does not quantify the actual amount of between-study variance.

Meta-regression is a regression analysis where you use the study as the unit of analysis. The main issue to be aware of here is the ecological fallacy - what applies at the group level may not apply at the individual level.

So, for both I2 and meta-regression, take care to interpret your findings carefully and correctly.

Need to read up on this further…

In meta-analyses, multiple smaller studies are combined to increase statistical power in order to estimate a treatment effect.
Ideally, all studies to be combined would be undertaken in the same way and to the same experiment protocols (e.g. study design, treatment regime, inclusion criteria, etc).
Differences between outcomes would then only be due to chance, and the studies would be considered homogenous.
But, this isn’t the real world. Studies are never done exactly the same.
Therefore, we expect some study heterogeneity arising from the variability in outcomes beyond that due to chance.
If heterogeneity is high, should the studies even be combined?
One (the most common?) measure of heterogeneity is \(I^2\).
\(I^2\) describes the percentage of variability in (treatment) effect estimates that is due to between study heterogeneity rather than chance.
- i.e. It tells us what proportion of the variance in observed effects reflects variance in true effects rather than sampling error.
It is a relative measure of heterogeneity and should NOT be interpreted as absolute.
- -> 0% - studies more homogenous (but the actual between-study variance may still be high - i.e. a small percentage of a large number may still be large).
- -> 100% - studies more heterogenous (but the actual between-study variance may still be low - i.e. a large percentage of a small number may still be small).
Meta-regression can be considered an extension of meta-analysis that combines, compares, and synthesizes research findings from multiple studies using regression analysis to adjust for the effects of available covariates on a response variable.
Covariates usually at the study level, rather than individual level (e.g. mean age, proportion men).
Usually underpowered.
Beware of aggregation bias -> ecological fallacy:
- What applies at the group level may not apply at the individual level.

On the 9th day of Christmas, a statistician sent to me:

Assess calibration of model predictions

Day 9 - Assess calibration of model predictions

This is another very specific issue that unless you’re doing prediction modelling, won’t really affect you in your typical day-to-day analyses.

Clinical prediction models are a special class of statistical model that are developed to inform the diagnosis or prognosis of a disease in patients. In other words, given a set of clinical variables, can we use that information to predict a patient’s individual risk of either having the disease (that’s diagnostic), or developing the disease down the track (that’s prognostic).

These models need to be developed in a very specific way to ensure the predictions are as reliable as possible and patients are not inadvertently harmed by giving them the wrong information. We want to mitigate the chance of both false positives and false negatives.

Model performance can be assessed primarily in one of three ways -

Discrimination, calibration and clinical utility.

Discrimination is routinely assessed and tells us about model accuracy - in other words can it correctly identify those with the disease from those without the disease.

However, calibration is NOT routinely assessed and the problem is that you can have a well-discriminating model that is poorly calibrated and that can harm patients.

So what is model calibration? In a nutshell, do the predicted probabilities of those with the disease that come from the model, align with the observed proportion of those in the sample with the disease. So, we feed data into our model, we estimate our model parameters and then we predict from our model. In a well-fitting model, we would find that the predicted probability of the disease would match with the proportion in the sample. So if 10% of people in our sample had the disease of interest, we would find that the model would also predict 10% of people with the disease. Models that are miscalibrated however, might under-predict (say 5% with the disease), or over-predict (say 20% with the disease) and neither of these scenarios are ideal for the patient when it comes to clinical decision-making. If the model is miscalibrated it may need to be reassessed using a different modelling approach - for example, penalisation or shrinkage methods.

Iwagami, M., & Matsui, H. (2022). Introduction to Clinical Prediction Models. Ann Clin Epidemiol, 4(3), 72-80. Alba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J. et al. (2017). Discrimination and Calibration of Clinical Prediction Models: Users’ Guides to the Medical Literature. JAMA, 318(14), 1377-1384. Van Calster, B., McLernon, D. J., van Smeden, M., Wynants, L., Steyerberg, E. W., & Topic, G. ’. E. D. T. A. P. M. O. T. S. T. R. A. T. O. S. I. (2019). Calibration: the Achilles heel of predictive analytics. BMC Med, 17(1), 230.

Clinical prediction models estimate outcome values (for continuous outcomes) or outcome risks (for binary or time-to-event outcomes) to inform diagnosis and prognosis in individuals.
In development and validation of such models, a full evaluation of model performance is critical to ensure predictions are correct and patients are not inadvertently harmed (due to false positives or false negatives).
For models that estimate outcome risk, performance can be evaluated in terms of :
- Discrimination (is model accuracy high? - ROC curves/AUC/etc)
- Calibration (do predicted probabilities align with observed proportions?)
- Clinical utility (how will this benefit patients?)
Discrimination is a relative measure of risk and is commonly assessed.
Calibration is an absolute measure of risk and is often ignored.
A well-discriminating model can be miscalibrated and this can affect its clinical utility.
e.g. Consider a model with a high AUC (area under the curve).
- Risk ratio may be correctly calculated as 5 - i.e a 5-fold increased risk of THE outcome in exposed vs unexposed groups.
- BUT, calibration is poor because absolute risks of the outcome are predicted as 5% in exposed and 1% in unexposed, when in reality the risks are 50% and 10%.
May need to reassess model using penalisation/shrinkage methods which reduce chance of overfitting.

On the 10th day of Christmas, a statistician sent to me:

Carefully consider the variable selection approach

Day 10 - Carefully consider the variable selection approach.

Data-driven variable selection methods - in other words selecting variables based on their statistical significance - is becoming increasingly frowned upon. And this applies to screening candidate variables in univariable models as well building multivariable models using approaches like forwards selection, backwards elimination, etc.

None of this is an issue if you prespecify one model and use that. When it becomes a problem is when you test many different models and present a ‘final’ model as if it were prespecified. This creates bias.

Standard errors and p values are too small and regression coefficients tend to be too large - so they’re inflated.

Look, I’m guilty of doing all of this myself, but its not like it’s a universally condemned practice. I was taught stepwise selection during my Masters and I believe it’s still being taught in statistics courses today. But I think there’s a greater recognition of the limitations of data-driven methods and an emerging consensus that we could all do things better.

And ultimately I think ‘best-practice’ in variable selection can be boiled down to two broad approaches.

For explanatory modelling, we should really be selecting our variables based on theory and our own expert knowledge - we can also the current literature as a crutch if we’re unsure. If we’re going to investigate causality then we really need to support our ideas with a DAG - a directed acyclic graph - and this basically shows all the theoretical relationships between the exposure, outcome and confounders in one diagram.

The point is that you pick one model and use that as is.

For predictive modelling, the approach is a little different - you can use more automated methods that rely on penalisation or shrinkage to come up with a final pool of variables. For this you’re mainly interested in maximising predictive power, so you tend to care less about p values anyway.

Thorpe, K. E. (2017). How to construct regression models for observational studies (and how NOT to do it!). Can J Anaesth, 64(5), 461-470. Heinze, G., Wallisch, C., & Dunkler, D. (2018). Variable selection - A review and recommendations for the practicing statistician. Biometrical Journal, 60(3), 431-449. Simms, L., Barraclough, H., & Govindan, R. (2013). Biostatistics primer: what a clinician ought to know–prognostic and predictive factors. J Thorac Oncol, 8(6), 808-813. Be familiar with penalisation/shrinkage.

There is increasing awareness of the disadvantages in using data-driven variable selection methods - i.e. selecting variables based on their statistical significance (p value). This applies in both:
- Univariable screening of ‘candidate’ variables.
- Multivariable model building - backwards elimination, forward selection, etc.
There is no issue if you prespecify a single model and then estimate that model from your sample of data.
The issue is when you test many different models and present a ‘final’ model as if it were prespecified.
This tends to result in standard errors and p values that are underestimated (too small) and regression coefficients that overstate their true relationship with the outcome (too large).
So what should we do? If we want best practice:
For explanatory modelling (associations/causation):
- Use theory, expert knowledge and the available literature to prespecify a model.
- In addition, analyses intended to claim causal effects should be supported by DAG’s (Directed Acyclic Graphs) where ALL potential confounders - and their relationships with the exposure, outcome and each other - are thoroughly documented.
For predictive modelling:
- Variable selection (through shrinkage) may be incorporated using methods such as lasso or elastic net, which start with a full model including all candidate predictors for potential inclusion.

On the 11th day of Christmas, a statistician sent to me:

Assess the impact of any assumptions

Have you considered sensitivity analyses?
What is a sensitivity analysis?
- A method to determine the robustness of an assessment by examining the extent to which results are affected by changes in methods, models, values of unmeasured variables, or assumptions.
A sensitivity analysis can help to answer the following questions:
- How confident can I be about the results?
- Will the results change if I change the definition of the outcome (e.g., using different cut-off points)?
- Will the results change if I change the method of analysis?
- Will the results change if we take missing data into account? Will the method of handling missing data lead to different conclusions?
- How much influence will minor protocol deviations have on the conclusions?
- What if the data were assumed to have a nonNormal distribution or there were outliers?
- What if the hazard ratio isn’t constant over time?

On the 12th day of Christmas, a statistician sent to me:

Use reporting guidelines to avoid overinterpretation

Day 12 - Use reporting guideline to avoid overinterpretation

This one is something we should all probably aim for more as well.

Doug Altman was a very-well regarded statistician who only recently passed on, but he once said “Readers should not have to infer what was probably done, they should be told explicitly”. So it’s incumbent on us as the researcher to make sure the reader can easily follow and understand what we’ve done, when we publish a paper.

And there are reporting guidelines available for most study designs to help you make your research as clear and transparent as possible. These are listed at the equator-network.org website and I’m sure you’ve come across these at some point already. Probably the STROBE guide for observational research is the more important one for us.

Look at STROBE and be familiar with it.

“Readers should not have to infer what was probably done, they should be told explicitly.”
Readers should not have to search for, or be unsure about:
- The rationale and objectives of a reported study.
- The study design.
- Details of the methods used.
- Participant characteristics.
- Results.
- Certainty of evidence.
- Research implications.
Make use of the appropriate reporting guidelines:
- Provide a checklist of items to be reported which represent the minimum detail required to enable readers to understand the research and critically appraise its findings.
Randomised trials:
- CONSORT (consolidated standards of reporting trials).
Observational studies:
- STROBE (strengthening the reporting of observational studies in epidemiology).
Systematic reviews:
- PRISMA (preferred reporting items for systematic reviews and meta-analyses).
Diagnostic test accuracy:
- STARD (standards for reporting diagnostic accuracy studies).
Prediction model studies:
- TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis).
All are available at https://www.equator-network.org